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A method for temporal Interpolation of an image 
sequence using object-based image analysis Is 
disclosed. The method comprises the steps of image 
segmentation for partitioning a Ichown image into 
objects, motion estimation, object-based motion field 
processing, determination of object depth order, 
detection of covered areas and newly exposed areas, 
adaptive interpolation using motion projection, and 
graceful degradation for alleviating remaining visible 
artifacts. The method produces interpolated images of 
high quality in real time for every type of image areas 
including fast moving objects, covered areas, and newly 
exposed areas. 
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(57) A method for temporal interpolation of an image sequence using object-based image analysis is disclosed. The 
method comprises the stq>s of image segmentation for partitioning a known image into objects, motion estimation, 
object-based motion field processing, determination of object deptli order, detection of covered areas and newly 

exposed areas, adaptive interpolation using motion projection, and graceful degradation for alleviating remaining 
visible artifacts. The method produces interpolated images of high quality in real time for evety type of image areas 
including fast moving objects, covered areas, and newly exposed areas. 
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Abstnet of tihe Disclosure 

A metfiod for tenq)oral inteipolation of an image sequence using object-b^ed 
image analysis is disclosed. The method comprises the steps of image segmentation for 
partitioning a known image into objects, motion estimation, object-based motion field 
processing, determiiiation of object dq>th order, detection of covered areas and newly 
exposed soeas, adaptive interpoliadon ilsdng motion projection, and jgracefui de^dation 
for alleviating remainil:^ visible ardficts. The metiiod produces interpolated images of 
high quality in real time for every type of image areas including fast moving objects, 
covered areas, and newly e;qx>5ed areas. 
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A Method for Temporal Interpolation of an Image Sequence Using Object-Based 

Imi^e Analysis 

Field of the Livention 

Tliis inveirtion relates generally to a mediod for temporal intetpolatioa of video 
signals generating one or moi^ images at any time instances between two known images. 
In particular, the mvention relate^ to a method for temporal interpplatiorL based on objeict« 
based unage analysis^ 

Background of the Invention 

Any kind of motion picture such as film, video or television is based on the 
inability of die human eye to distinguish between separate images of an image sequence 
if the images are presented at a rate higher than approximately 16 ims^es per second. 
There are numerous standards for fihn, video or television having different rates of 
images per second called fiame rate. In order to di^lay a motion picture produced with 
standard A using standard B the motion picture has to be converted. For example, in 
order to convert a video signal from a frame rate of 50 Hz to a frame rate of 60 Hz or 
vice-versa temporal interpolation is needed because most time instances where images 
have been taken do not coincide. 

Furthermore, transmission of video signals with a high frame rate is often limited 
by the capability of existing communication channels, for example in video conferencing 
and video telephony. Therefore, images are dropped at the encoder prior transmission 
with a low bit rate and Iiave to be reconstructed at the decoder. 

A first method for im^e i«constmction was to repeat a nearest available im^e, 
resulting in jerkily moving objects. 

In other early temporal interpolation methods, motion of objects in an image 
sequence was not taken into account. A pixel value was interpolated as a linear 
combmation of values of pixels of two known images havmg same spatial coordinates. 
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These methods resulted in htteipolated images of poor quality within areas that are in 
motion. 

More recently, advanced methods are taking motion into £»:count in order to 
preserve a natural impression of motion. Such methods are called "motion-compensated 
temporal interpotetion methods'\ Motion-^omp^ated methods a^^^^ 
fbllovwng lefereiices, which are hereby incorporated by 
U.S. Patent No. 4,65 1 ,207 issued March 17, 1987 to Bergmann et al.; 
U.S. PatentNo. 4,771,331 issued September 13, 1988 to Bierling et al.; 
U.S. Patem No. 5,214,751 issued May 25, 1993 to Robert^ 
U.S. Patent No. 5,394,196 issuedFc^^ 
U.S. Patent No. 5,508,747 issued April 16, 1996 to Lee; 

Peter Csillag and Lilla BdiOczky, "MC Frame Interpolation Applyii^ Mo]li9n-Basfe4 
Segmentation and an Accelerated Motion Model", PCS, Marph 1 996; and, 
Tien-ying Kuo and C.-C. Jay Kuo, "Motion -Compensated Interpolation for Low-Bit- 
Rate Video Quality Enharicernient^ SP^^ 

These methods comprise the st^s of motion estimation, motion field seglthentatioh, aind 
adaptive interpolation. With motion estimation, each pixel is associated with a motion 
vector. The vector field is then segmented mto finir types of regions: stationary regions, 
moving regions, covered regions, and newly exposed regions. The segmentation is 
carried out by classifying the estimated motion vectors. Pixel values of an image to be 
interpolated are determined using an adaptive interpolator together vrith the region and 
motion information. These rnetiiods produce interpolated images of good quality only if 
the estimated motion vectors are consistent with tiie true motion of objects and if the 
motion of the objects is not fast. 

However, these motion compensated temporal interpolation methods often 
produce visible artifacts in covered areas, newly exposed areas^ and areas with fast 
motion. The artifacts are caused by pixel-based image analysis used in these methods to 
segment images or motion vector fields, that is, each pixel is classified according to the 
estimated motion vector of the pixel. The estimated motion vectors in covered and newly 
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exposed areas are usually erroneous because Hiese areas have no counterpart in one of the 
known images. Within moving objects, some estimated motion vectors may be different 
from the true motion vectors due to image noise, shadow, or lighting change. The 
erroneous motion vectors result in erroneous classification of the associated pixels 
leading to artifects. It is well known in the art that segmentation based on motion vector 
fields cannot accurately define moving object boundaries even if the motion vector field 
within moving objects is well estimated. 

It would be advantsfgeous to provide a method for temporal interpolation avoiding 
visible artifacts. Therefore, it is an object of the inventi^^ 
temporal mt^rpolation using two knomi images 
analysis in order to determine erroneous motion vectoi^. 

It is a further object of the invention to provide a method that provides 
inteipolated ims^es of Mgji quality for every typ 

It is yet another object of die invention to prpvide a method that provides 
inte]i]|Qlated ixiui^^ 

Summair]^ of the Inven^^ 

In accoidance with the invmtion there is provid^ a method for generating 
intecpolated images of high quality for every type of im^e area. It is an advantage of tiie 
present invention to reliably dfstermine erroneous motion vectors due to use of object- 
based image analysis, which substantiaUy reduces visible artife^^ 
nev«dy exposed areas, and fast moving areas. 

In accordance with the invention there is provided, a niethod for generating an 
image firom at least two known images of an image seqixence conaprising tiie s 

a) segmenting at least one of the at least two known im^es into objects, the objects 
haying a homogeneous interior with respect to luminance and colour; 
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b) estimating motion from the at least one segmented known image towards anotiier of 
the at least two known images and assigning a motion vector to each pixel of the at least 
one segmented known image generating an estimated motion fiel(it 

c) proceissihg the estimated motidn field to obtain a smooth motioli field within each 
object; and, 

d) calculatingpixel values of the image to be interpolated 
projection* 

In accordance with one embodiment of the present invention there is provided, a 
method for temporal interpolating an iniage from two known images of an image 
sequence comprising the steps of: 

a) segmenting one of the two known images which is closer to tihe image to be 
mterpolated into objects^ the objects having a homogeneous interior with respect to 
luminance and colonic 

b) estimating motion from the segmented known image towards the other known image 
and assigning a motion vector to each pixel of the segmented known image generating an 
estimated motion field; 

c) processing tte estimated motion field to obtain a smooth motion field within each 

object;" ^ ■ . . . • ■ 

d) determining object-depth order and detecting covered areas; 

e) calculating pixel values of the image to be interpolated using object-based motion 
projection; and, 

f) alleviating remaining visible arti&cts by eliminating false sharp edges through gracefiil 
degradation. 
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In accordance with another emtK>diment of the present invention there is 
provided, a method for temporal inteepplaling an image from two known images of an 
image sequence comprising the steps of: 

a) segmenting die first and the second known image into objects, the objects haying a 
homogeneous interior with respect to luniinance and colour, 

b) estimating motion from the first segmented known image towards the second 
segmented known image and assigning a motion vector to each pixel of the first 
segmented known image generating a first estimated motion field; 

c) estimating motion firom the second segmented known image towards the first 
sqgmrated known image and assigning a motion vector to each pix 
s^mented known image goierafing a second estimated motion field; 

c) processing the first and second estimated motion field to obtain a smooth first and 
second motion field within each object; 

d) determining object^lepth order and detecting covered areas for the first and second 
motion field; 

e) calculating pixel values of a first image to be interpolated based on the first motion 
field using object-based motion projection; 

f) calculating pixel values of a second image to be interpolated second motion field using 
object-based motion projection; 

g) alleviating remainiiig visible artifacts in the first interpolated image by eliminating 
&lse sharp edges through gracefiil degradation; 

h) alleviating remaining visible artifacts in the second interpolated image by eliminating 
false sharp edges through graceful degradation; and, 

i) detemuning the image to be interpolated as a weighted average of the first inteipolated 
image and the second interpolated image. 
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Brief Description of the Drawings 

An exemplary embodiment of the invention will now be discussed in conjunction 
with the attached drawings in which: 

Fig. 1 illustrates schematically covered areas and newly exposed areas; 

Fig. 2 is a block diagram illustrating^^heinatically a method for t^poral interpolation of r 
an image according to flie invention; 

Fig. 3 illustrates schematically segmentation of a known image and an estimated motion 
vector using the method of temporal interpolation according to the invention; 

Fig. 4 illustrates schematically erroneous motion vectors estimated in a covered area; 

Fig. 5 illustrates schematically an object-adaptive interpolator for correcting erroneous 
motion vectors within an object according to the present inventi^ 

Fig. 6 illustrates schematically i^lication of an object-adi^ve filter for smoothing the 
motion field within an object accoxding to the present iiiventib 

Fig. 7 illustrates schematically motion projection of objects for determining object depth- 
order and covered areas according to the pissent mven 

Fig. 8 illustrates sch^natically uiteipolation of an image containing a &st moving object 
in front of a stationary background; 

Fig. 9 illustrates schematically projection of a pixel in a known image to a point between 
four pixels in ah image to be interpolated; 

Fig. 10 illustrates schematically creation of a false edge between a newly exposed area 
and other interpolated areas; 
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Fig. 1 1 illustrates schemadcally detection of a band surrounding a newly exposed area 
according to the present invention; 

Fig. 12a is a block diagram illustrating schematicaliy an embodiment of a method for 
temporal interpolation of an image according to the present invention; 

Fig. 12b is a block diagram illustmting schematically another embodiment of a method 
for temporal interpolation of an image according to the present invention; and 

Fig. 12c is a block diagram illustrating schematically yet another embodiment of a 
metitod for teiiqporal interpolation of 

Detailed Descriptkiii 

Referring to Fig. 1 two knowin images 1 and 2 of an image sequence are shown. 
For example, an image can be a field or a frame of a video signal. The known images 1 

and 2 correspond to time instants /^and /2 with /, < ti , and are deno^d by /{r,f,)and 
/(r,/^) , respectively, wherein r = [jK:,>>]^represeiits spatial coordinates of image pixels. 
From the two known images 1 and 2 an image to be int^olated l{pyt) 3 at time instant 
r with /] < / < ^jis then detomined iising the method according to the invention which 

will now be described wherein p == [x,>']^represents spatial coordinates of image pixels 
within the image to be interpolated. 

The method for temporal interpolation according to the present invention, as 
shown in Fig. 2, comprises the following sections: 

1) image segmentation for partitioning one of the known images into objects, 
based on image features such as luminance, colour, and/or texture; 



7 



. GA 02279797 1999-08-06 

Doc. No. 18-23 CA 



Patent 



2) estimation ofthe motion from tiie segments kno\m in^ 

known image and assigning a motion vector to each pixel of fhe segmented 
known image; 

3) object-based motion-field processing fpr dcstectiiig and correcting enpneous 
naption vectois md for spioollu^ 

4) determination of depth-order, covered areas, and newly exposed areas by 
projecting objects in fhe segmented known image to the other known image; 

5) adaptive int^rpolaftion for calculating each pixel value of the interpolated 
image using infppqtation about motion, object d^t-or<)er^^ ar^^ and 
ne^y exposed areas; and, 

6) graceiful degradation for makin r^maiiijng artifacts to the human 
eye. . ■. ■ 

in section 1) one of fhe known images 1 or 2 is segm^ted into objects* For the 
following description it is assumed that the known image 1 is segmrated, as shown in 
Fig. 3. In fhe method according to tiie invention an object 11 is defined as an arbitrainly 
shaped region satisfying the foUowing two conditions: 

(1) its interior is homogenous with resfpect to image features such as luminance, 
color; andA)r texture; and, 

(2) its contour coixesponds to abrupt changes of fhe above features. 

The method for image segmentation into objects used in the method according to 
the invention is disclosed in a paper by one of the inventors, D. Wang, " A multiscale 
gradient algorithm for image segmentation using watersheds", published in Pattem 
Recognition, Vol. 30, No, 12, pp. 2043-2052, 1997. Other methods for image 
segmentation into objects may be used if the resulting objects satisfy the above 
conditions. 
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The second step - section 2) - for interpolating image l{pj) 3 comprises motion 

estimation from image /(r,r,) 1 to im^e /(r./j)!, and to assign a motion vector 15 to 

each pixel 17 of image ^r,r, ) 1. For the motion estimation any pixel-based, block-based, 
or object^bascfd algorithbi may be used. Furtfieniibre, motion fields of previous images 
may also be used for tietter estimating the inotioh field at the expense of additional 
computixig time and memoxy. The selection of a motion estimation dgoritiim has great 
influence on bptii the quality of inteipolated images and the computational cost for the 
interpolation. In the method according to the invention a hierarchical block-matching 
algorithm i^ used, llie hieranJiic^sdbl^^ [»x>diubes motion 

fields of good qiiality, alkmiiig ii^^ 

The estimated motion field obtained in section 2) is not smooth within an object, 
especially whin a pixel4»ased or block-based motion estimationMgorithrn^^ been used, 
due to erroneous motion vectors. An moneous motion vector may result firom occlusion 
of objects or newly exposed areas in successive images. For example. Fig. 4 sliows t^o 
successive images wherein a rectangle 22 is moyiiig witii resspect to a stationary 

background 24. The first image /(r.^i) 1 is segmenlted into two objepts: the rectangle 22 
and the background 24. The motion field of an area 26 left to the rectangle 22 in the first 

image /(r,/,) 1 cannot be correctiy estimated because the rectangle 22 overiaps tiie 

corresponding area in the second im^e /(r,/2)2« resulting in erroneous motion vectors 
28. The erroneous motion vectors 28 are disturbing the estimated motion field causing a 
"noisy" motion field. A noisy motion field may result in an interpolated image of poor 
quality. In reality^ true motion fields are smootii within each object. For example, a true 
motion field is zem within the stationary background 

According to the present invention the estimated motion field obtained in section 
2) is processed in order to obtain smoothness within objects. The method for object-based 
motion field processing according to the invention - section 3) - comprises the following 
steps: 
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(a) The estimated motion field within an object is first approximated using a 
motion model resulting in an approximated motion field . The motion model 
describes a smooth motion field within the object. In the method according to the present 
invention an affine motion model with 6 parameters has been used for approximating the 
estunated motion field. An approximation error is defined as ||«{r) - ii(r)| . In general, a 
small number of motion vectors within the object are erroneous because of occlusion or 
other reasons. Hence, the approximation error is small for correctly estimated motion 
vectors, whereas it is large for erroneous motion vectors. An estimated motion vector is 
classified as an erroneous motion vector if its approximation error is greater than a 
threshold . Based on test results the two components - in x and y direction - of the 
threshold 3^ are set as: 



vsdierein and are average absolute approximation errors of tiie two motion vector 
compommts in x and y direction, rei^^ 

(b) Erroneous motion vectors are replaced using an object-adaptive interpolator. 
The object-adaptive interpolator uses correctiy estimated motion vectors belonging only 
to an object 11 under consideration, as shown in Fig. 5. The interpolator performs a linear 
interpolation w, in y- direction using two correctly estimated motion vectors and 
i^wMch are closest to the erroneous motion vector u . Then another interpolation iVj is 
performed in x-direction using two correctiy estimated motion vectors % and 1/4 closest to 
the vector u . Finally, the erroneous motion vector u is replaced by a weighted average of 
the linear interpolations >i^|and W2 . With c4 denoting a distance between 2/and for 
0 < ^ < 4 , the object adaptive interpolator is expressed as follows in equations (la) - 





4eyjf 4ey > 3 
^otherwise 



(Ic): 
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(la) 



^2 = 



(lb) 



(ic). 



If, for e>^ple, only one of tbe correct motion vectors and i/jis available 
because the erroneous motion vector is adjacent to anoflx^r object is equal to the 
available motion vector as is obvious &om Fig. S and equation (la). If none of the vectors 
Ui and ff2 is available u is equal to . In case no correctly estimated motion vectors are 



available u is left undefined and will be determined after ftuther processing. 

After each erroneous motion vector for wiiich at least one of Uj^t^^i/jor U4is 
available has been interpolated, erroneous motion vectors for which none of u^^Uz^u^ or 
1/4 is available are interpolated using previously interpolated motion vectors, 

(c) The motion field within an object is then further smootfaened using an objectr 

adaptive filter. The filter uses a window 32, which is moved to every pixel of an object 

11 such that its center coincides with the pixel 34, as shown in Fig. 6. At each pixel 

position the filter is used to calculate an average value of the motion vectors within the 

window 32 and belonging to the object 11. The average value is then assigned to the 

central pixel 34 as the final motion vector. The window size of the object-based fillter is 

« 

determined based on the unage size. ^With D being the greater of image width or image 
height appropriate window sizes have been found fix>m test results using the method 
according to the invention as follows: 

11x11 pixels, for D < 356 pixels; 

15 X 15pixels, for 356 S £> < 1024 pixels; 

19 X 19pixels, for D > 1024 pixels. 
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Object*based motion field processing according to the present invention is highly 
advantageous by providing a metfiod for correcting erroneous motion field vectors. The 
method takes advantage of the object-based segmentation of images which allows 
processing of the motion vectors within each object of an image and correction of 
erroneous motion vectors. Prior art methods such as motion-compensated temporal 
interpolation methods are not capable of correcting erroneous niotipn vectors because 
these methods use estimated motion vectors to segment images. Erroneously estimated 
motion vectors result in incorrect segmentation, and the incorrect segmentation cannot be 
used to correct erroneously estimated motion vectors. Ahotiier cldvanti^e of the object- 
based temporal inteipolation of images according to the invention will become evident in 
the detenrnxmtion of object depth-order within int^polaled imi^s. The depth-order of an 
object refers to a distance between the object in a real scene and a camera capturing the 
object When objects are moving the depth-order of objects determines appearance and 
disappearance of the objiects in the ca^^ 

Referring to Fig. 7, a method for determining the depth-order according to tiie 
invention - section 4) - will now be described. Every object within a first known image 

/(r,/|)is projected to a second known image /(r,/2)according to its motion field. With 
and Rj , for ij = 0,1^, » denoting the objects m the first image l{r,t{) and 

zi(r) denoting the motion field after object-based motion field processing the projection of 
the object is determined as: 

iJ = {r + «(r)|r€J^}. 

Having determined the projection of the object a projection error for each 
pixel havitig coordinates r is tiien defined as |/(r,/|) /(r + »(r),/2^^^^ 

If Ff overlaps another projection Fj as shown in Fig. 7 then objects and Rj have 

different depths. The overlapping area O^j in the second image /(r^/j) comprises either 
overlapping area AgOf object R^ or overlappmg area Aj of object Rj . If O^j comprises 
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i^^then object is in j&ont of object Rj. In this case a projection error of area is 
smaller than a projection error of area Aj . Therefore, the depth-order of objects and 
covered areas is determined by comparing the projection errors of these areas. The total 
projection errors of areas ^ and are defined as: 

If < ^Aj^ ^ the overlapping area Oij corresponds to area ^and, therefore, 
object J!^is in fiont of object J?yand is a covered area. This infonnatipn is reprinted 
by assigning a lower depthH>rder to object and a higher dep^ . 

Shape information of overlapping areas such as Of^ is stored in an image or 
mask- S(r) . A pixel of S(r) is equal to " 1 " if it belongs to a cov«ried area, otherwise it is 
equal to "^O**. The mask 5(r) will then be used in a following step of ada{)tivd 
interpolation since part of a covered area may appear in an image to be interpolated. 

If a large area of an object is covered in the second known image the motion field 
within the object is not reliable. This results from estimated motion vectors in a covered 
area being usually erroneous and step (a) of the object-based motion field processing 
bemg not effective in detecting a large number of erroneous motion vectors. Therefore, if 
the ratio of the covered area to the total area of the object is larger than a threshold, the 
motion field within the object is replaced by a spatial interpolation with motion fields of 
adjacent objects. In tests of preferred embodiments of the metiiod according to 
invention a tiireshold of about O.S has been found to provide good results. The spatial 
interpolation is similar to that described in step (b) of the object-based motion field 
processings except that it is not object-adaptive. For each motion vector uof an object 
having a ratio of covered area to total area larger than the threshold, the interpolator 
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calculates equations (la) - (Ic) using the four closest motion vectors of adjacent objects 
in X and y diiection and replaces the motion vector ii with the result 

The method for det^inining depth-prder and covered areas according to tibe 

invention is highly advantageous over the prior art. If two pixels of sai image /(r,/, ) are 
projected to a same point in the image to be interpolated, prior art methods classify tihe 
pixel having a larger projection error as a covered pix^l. The classification in prior art 
methods is based on individual motion vectors. Therefore, prior art methods are less 
reliable as can be sem iii an example shown in Fig. 8. Two motion vectoi^, one 
originating from a fast moving object and another originating from a stationary 

background within a first image l{rJi) , are well estimated and projected to a same point 

in the iroage to Mint^ipol^^ 

firom the fast moving object has a largesr projectipn error flum the motion vector 
originating fiom the stationary background leading to a wrong classification which 
results in visible arti&cts disturbing the fast moying object The method of the present 
invention avoids this problem by, firstly, using a projection to the second known hn^e 

/(r ) and, secondly, comparing the total projection errors within areas instead of 
individual pixels. 

Pixel values of the image to be uiterpolated ) ai^ then calculated using 
adaptive motion projection - section 5). A pixel /('',A ) within the first image is 
projected to a point r + cas^r) within the iitiage/^p,/ j to be interpolated, wherein 
a = (r - /, ) / (^2 ^ A ) is * motion vector associated with the pixeL However, 

point r + au{r) may not correspond to a pixel position within the image /(p,/ j . A 

strai^tforward solution to this problem is that a pixel closest to point r + aii(r) is 
mtexpolated with the motion vector u(r) . A major drawback of this solmion is that spme 

pixels of the image to be interpolated l{^p,t ] may be missed resulting in visible artifacts. 
In order to minimize the number of missed pixels, the four pixels nearest to point 



14 



Doc, No- 18-23 CA 



CA 02279797 1999-08-06 



Patent 



r + coir) are inteipolated with the motion vector «(r) , as shown in Fig. 9. The adaptive 
intexpolation is then carried out as follows: 

If pixel /( A^) ^ not been previously interpolated, ttie value of the pixel and its 
interpolation enror are detennined by: 

l{pj) fil{p- o^ (2b) 

wift fi^\-a. Equation (2a) is used when an area is covered - S(r) » 1 - , that is 
the pixel value of the first image is projected to the interpolated knage. Otherwise, the 
pixel vakie is int^OP^l^^ 
and the second image, llie inteipolationenxtri 

In addition, the depth-onl^ of object is stored^i^^^^ image denoted by F(p)and 
the motion vector i^r) is stored in a projected motion field denoted by {/(p) ^ u{r) . 

If pixel /(a^) has been {Mreviously interpolated by projecting otiier pixels, tiie 
depth-order of object J?, is compared with that stored in i^p) . If the depth-order of Rf is 
lower, the old values of l{p>t) , ^(/>) > J^p) an4 are replaced by the new values. If 
the depth order of i^isthesanieas J^/?), the new projection error is compared 
with the old one. If the new error is smaller, the old values of /(a^) » ^{p\ » 
(/(z?) are replaced by the new ones. Otherwise, the old values are retained. 

Newly exposed areas in the second known image l{ryt^ do not receive any pixel 
projection firom the first known image , as shown in Fig. 1. Newly exposed areas 
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may appear in the image to be interpolated l{pj) . ln order to calculate a pixel value 

within the newly exposed areas in the image to be interpolated /{/>, t) , the projected 

motion field u(p)is interpolated u$ing the spatial interpolator described in the section of 

the motion field processing. The values of r) and E{p) ia these areas are tben 
detemiined as ibllows: 

lij>,i)^j{p-^^P^ (4) 

= (5) 

Transition between a newly e?!^$ed areasf and other iiiterpola^ areas is not 
smooth even if ti^ese areas belong to a same object TMs results :^m the fiict diat in 
natural im£^es the transition from one object to another is not a sudden change but a 
transition zone between two adjac^t objects. A good image segmentation method 
partitions objects along the middle of transition zoxies. The transition zones are also 

projected to die interpolated image l{pf t) . The project^ transition zones create fiilse 

edges in l{p»t) and make the transition between newly exposed areas and other areas not 
smooth, as shown in Fig. 10. In a method according to the invention this problem is 
solved by modifying pixel values within a band surrounding every newly exposed area 

Qin the interpolated image . Referring to Fig. 1 1 , a band around a newly exposed 

area Q is detected by Q^B-Q, wherein ® denotes dilation and B a structuring 

element. If tiie interpolation error of a pixel witiiin the band is larg^ tiian a 
predetermined threshold, the value of the pixel is replaced using equation (4). Otii^rwise, 
die value remains uiichanged. Tests using this method produced good resulte for a 
structuring element of 5 x S pixels and a threshold of 30. This method according to the 
presmt invention produces a substantially smootii transition between newly exposed 
areas and other interpolated areas. 
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The method for interpolating images accpiding to the present invention [produces 
inteipolated images of high quality for most unage sequences. However, there are 
situations where some visible arti&cts may still occur such as defonning objected lifting 
changes between two known unages arid very fiist moving objects in video signals having 
a very low frame rate. A visible arti&ct is iraiaily a small region having an incorrect 
colour seen as a false sharp edge surroundmg an object or parts of an object A method 
acGordii^ to the present invention to alleviate such remaining visible artifacts - section 6) 
- is based on the characteristic that the human visual system is less sensitive to blur pf an 
object than a Mse sharp edge of tiie object, especially when the object is moving fast. To 
that end, the method according to the present invention makes such artifact^ less| visible 
to the human eye by eliminating false sh^ edges through graceful degradation. If the 

motion vector L^(/7)of a pixel wtl&tiie i 

predetermined threshold the value of the pixel is replaced by an average value within 
an £ X X window centered on the pixel. The threshold and the window size are 
determined depending on image size. With D denoting image width or height whichever 

is greater^ dd&aoting the module of the motion vector u{p) » a threshold of T„ ==p / 10 
and a window size L^dl lOhave beam foimd to produce good results. 

Refeiring to Fig. 12a a flow diagram of an embodiment of the method for 
temporal interpolating of an image according to the invention is shown. A first known 
image is segmented for partitioning the iixi^e into objects. Motion from the first image 
towards the second image is then estimated and a motion vector is assigned to each pixel 
of the first image. The order of the following sections remains the same as in the outiine 
of the method shown in Fig. 2. 

Interpolated images of better quality are pbtained using another embodiment of 
the method according to the present invention, shown in Fig. 12b. In this preferred 
embodiment the known image which is Qloser to the image to be interpolated is 
segmented into objects. Interpolated images are of better quality using this embodiment 
because the known image closer to the image to be interpolated has a higher correlation, 
therefore, increasing accuracy. 
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Another prefened embodimwit produces two interpolated images r(p,t) and 
/"(/>,/) , as shown in Fig. 12c. Image r{pj) is generated with segmentation of the first 
known image and motion estimation from the first known image towards the second 
known image, v^eieas is generated with segmentation of the second known 

image and motion estunation from tibe second known hnage towards the fir^ knoym 
image. Remaining visible artifacts in the interpolated images /*(p»r) and A 0 ^ 
alleviated then dirough gracefid degradation of false sharp edges before generating die 
fmal interpolated image. The final interpolated image is a weighted average of 

/•(/>,/) and described as follows: 

..with .. ^. ^ 

(2 + £'(/>)X/ -r.) 
^ = (2 + ^'(p)Xr-0V(2 + £"(p)X'. -0 ' 
wherein £'(p,^)and ilE"{Ar) are interpolation mors of and 
resfpectively« This method usually provides interpolated images of even better quality but 
requires substantially more computing effort in oider to obtain a final interpolated image. 

Alternatively* the final interpolated image is generated without alleviating 
remaining visible artifacts in the interpolated images r{p,t) and /"(aO • P^se sharp 
edges in the final interpolated image are then eliminated through graceful degradation. 

Furthermore, in yet another embodiment of the method for temporal interpolating 
of an image according to the invention more than two known images are used for motion 
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estimation and using higher order inteipolation algorithms in crdi^ to increase accuracy 
of images to be interpolated. , 

The method for temporal int^polation of an image according to the present 
invention produces intexpolated images of hij^ quality and allov^ generaidon of 
intopolated images in re^ time. Iliic^fi>re, itie mdfaod isKsconi^ to the preseixt 
is highly advantageous over the prior art and^ fiirttennbre, allo^ to overcome bandwidth 
problems in applications such as video conferencing or telept^esence by capturing and 
transmitting video signals at a lower frame mte and generating inteacpolated images for 
display in real time. Furthermore, the high accuracy of the methpd according to the 
invention makes it ideal for any kind of conversion of image sequences from one 
standard to another, for example, conversion between standards having a different fi^e 
rate; conveision between interlaced and non-interlaced or conversion of any image 
sequence to the new HDTV standard. Yet another application of the method accoidihg to 
the present invention is spatial inteipolation of stereo iroages. 

Of course, numerous other emlxxfiments may be envisaged Withbut departing 
from the spirit and scope of the invention. 
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Claims 

What is claimed is: 

1 . A method for generating an image from at least two known im^es of an irnage 
sequence comprising the steps of: 

a) s^menting at least one of the at least two known images into objects, the objects 
having a homo£;eneous interior with respect to luminance, colour, and/or texture; 

b) estimating motion from the at least one segmented known image towards another of 
the at least two known images and assigning a motion vector to each pixel of the at least 
one segmented known image generating an estimated motion field; 

c) processing the estimated motion field to obtain a smooth motion field within each 
object;and, 

d) calculating pixel values of the image to be inteipolated using object-based motion 
projection. 

2. A method for generating an image from at least two known images of an image 
sequence as defined in claim 1, wherein the image sequence comprises a video signal. 

3. A mediod for generating an image from at least two known im^es of an image 
sequence as defined in claim 1, vtiserein the image is generated as temporal interpolation 
from die at least two known images. 
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4. A method for generating an image fix>m at least two known images of an image 
sequence as defined in claim 1, comprising the step of: 

determining object-depth order and detecting covered areas. 

5. A method for generating an image from at least two known images of an image 
sequence as defined in claim 1, comprising the step of: 

alleviating remaining visible artifacts by eliminating false sharp edges through gracefiil 
degradation. 

6. A method for generating an image fiom at least two known images of an imkge 
sequence as defined in claim 1, wherein step c) comprises the steps of: 

approximating the estimated motion field within each object using a motion model 
describing a smooth motion field within each object; 

companx^ the estimated motion field vnUi die approximated motion field to determine an 
approximation error for each motion vector of the estimated motion field; and, 

classifying the motion vectors of the estimated motion field into correctly estimated 
motion vectors and erroneous motion vectors by comparing the approximation error of 
each estimated motion vector to a threshold. 

7. A method for generating an image from at least two known images of an image 
sequence as defined in claim 6, wherein the motion model is an afiSne motion model. 

8. A method for generatmg an image from at least two known unages of an image 
sequence as defined in claim 6, comprising the step of: 
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replacing each erroneous motion vector with a motion vector obtained by interpolating 
correct motion vectors closest to the ^oneous motion vector using an object-adaptive 
interpolator, the correct motion vectors belonging to a same object as the erroneous 
motion vector. 



9. A method for generating an image from at least two known images of an image 
sequence as defined in claim 8, comprising the step of: 

smoofhmg the motion field within each object using an object-adaptive filter* 

10. A method for generating an image from at least two known infixes of an image 
sequmce as defined in claim 9» wherein the object-adqytive filter comprises a window for 
calculating an average value of the motion vectors within the window and beloi^ing to a 
same object and for assigning the average value to die motion vector of a pixel centered 
inthe window. 



11 . A method for genemting an image fipm at least two known images of an image 
sequence as defined in claim 10, wherein the window is moved such that each pixel of an 
object is centered. 

12. A method for gen^ting an image from at least two known images of an image 
sequence as defined in claim 4, comprising the steps of: 

projecting every object of the segmented known image to another known image and 
calculating a projection error for each pixel of the segmented known image; 

calculating a total projection error for each of two overlapping areas; 
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comparing the total projection errors of overlapping areas and assigning a depth-order to 
each of the objects which contain the overlapping areas. 

13. A method for generating an image from at least two known images of an image 
sequence as defined in claim 12, comprising the steps of: 

determiniiig covered areas in dependence iqK»n the depA^ 

generating a mask containing shape information of the covered areas. 

14. A method for generating an image from at least two known images of an image 
sequence as defined in claim 12, comprising the steps of: 

modifying the motion field within objects comprising a large covered area by replacing 
the motion vectors within the object with motion vectors interpolated fix>in motion 
vectors of acQacent objects. 

15. A method for gen«:ating an image from at least two known inaages of an image 
sequence as defined in claim 1, wherem step d) comprises the step of: 

calculating pixel values of the image to be int^polated as a weighted average of pixel 
values of the segmented known image and the at least other known image. 

16. A method for generating an image from at least two known images of an image 
sequence as defined in claim 12, comprising the step of: 

retaining one projection in areas where projections overlap each other according to the 
depth-order of objects. 
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17, A method for generating an image from at least two known images of an image 
sequence as defined in claim 16, wherein the pixel values witfiin projected areas 
corresponding to covered areas are equal to the pixel values of the segmented known 
im£^e. 



18. A method for genemting an image from at least two known images of an image 
sequence as defined in claim 16, wherein the pixel values within projected areas 
corresponding to newly exposed areas are determined using only the pixel values of the 
other known image. 

19. A method fi>r generatmg an image fiom at least two known unages of an image 
sequence as defined in claim 1 8, comprising the step of: 

determining pixel values for pixels within a band surroimding newly exposed areas using 
only the pixel values of the other known image if an interpolation error of pixel values of 
the pixels within the band obtained as a weighted average of pixel values of the 

segmented known image and the at least other known image is greater than a 
predetermined threshold. 



20. A method for generating an image from at least two known images of an image 
sequence as defined in claim S, comprising the step of: 

comparing motion vectors of each pixel of the genemted image with a threshold; and, 

replacing a pixel value of a pixel having a motion vector greater than the threshold with 
an average value of pixels within a window centered at the pixel. 
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21. A method for temporal interpolating an image firom two known images of an image 
sequence comprising the steps of: 

a) segmenting one of the two known images which is closer to the image to be 
interpolated into objects, the objects having a homogweous interior with respect to 
luminance and colour; 

b) estimating motion ftom the segmented known image towards the other known image 
and assigning a motion vector to each pixel of tfie segmented known image generating an 
estimated motion field; 

c) processing the estimated motion field to obtain a smooth motion field within each 
object comprising the steps of: 

approximating the estimated motion field within each object using an affine 
motion model describing a smooth motion field within each object; 

comparing the estimated motion field with the approximated motion field to 
detemiine an approximation error for eac^ motion vector of the es^ 

classifying the motion vectors of the estimated motion field into correctly 
estimated motion vectors and erroneous motion vectors by comparing the 
approximation enor of each estimated motion vector to a threshold; and, 

replacing each erroneous motion vector with a motion vector obtained by 
interpolating correct motion vectors closest to the erroneous motion vector iising 
an object-adaptive interpolator, the correct motion vectors belonging to a same 
object as tise erroneous motion vecton 

d) determining object-depth ord^ and detecting covered areas comprising the steps of: 

projecting every object of the segmented known image to another known image 
and calculating a projection error for each pixel of the segmented known image; 
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calculatmg a total projection CTt>r fore^ 

comparing the total projection errors of overlapping areas and assigning a depth- 
order to each of the objects wdiich contain the overlapping areas; 

determining covered areas in dependence upon the depth-order; and, 

generating a mask containing shape information of the covered areas; 

e) calculating pixel values of the ima^e to be interpolated using object-based motion 
projection; and, 

f) alleviating remaining visible artifacts by eliminating false sharp edges by replacing the 
value of a pixel within the interpolated image with an average value within a window 
centered on the pixel if the motion vector of the pixel is greater than a threshold. 

22. A method for temporal interpolatii:^ an image from two known images of an image 
sequmce comprising the steps of: 

a) segmenting the first and the second known image into objects, the objects having a 
homogeneous interior with respect to luminance and colour; 

b) estimating motion fix>m the first segmented known image towards the second 
segmoited known image and assigning a motion vector to each pixel of the first 
segmented known image goiemting a first estimated motion field; 

c) estimating motion from the second segmented known image towards the first 
segmented known image and assigning a motion vector to each pixel of the second 
segmented known image generating a second estimated motion field; 

c) processing the first and second estimated motion field to obtain a lanooth first and 
second motion field within each object; 
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d) detemuning object*depth order and detecting covered areas for die first and second 
motion field; 

e) calculating pixel values of a fii^ image to be interpolated based on the first motion 
field using object-abased motion projection; 

f) calculating pixel values of a second image to be interpolated second motion field using 
object-based motion projection; 

g) alleviating remaining visible artifacts in the first in^rpolated image by elindnating 
fiilse shaip edges flirough graceful degradation; 

h) alleviating remaixiing visible artifacts in die second interpolated image by eliminating 
fidse shaip edges tlm>ugh graceMd^gradation;^^a^ 

i) determining the image to be interpolated as a weighted average of the &st interpolated 
image and the second interpolated inu^e. 

23. A method for temporal interpolating an image from two known images of an image 
sequence as defined in claim 22, wherein weighting coefficients in step i) are determined 
in dependence upon interpolation error and time distance between the two known images 
and the interpolated image 
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